MR2CPPIS: Accurate prediction of protein-protein interaction sites based on multi-scale Res2Net with coordinate attention mechanism.

Proteins play a vital role in various biological processes and achieve their functions through protein-protein interactions (PPIs). Thus, accurate identification of PPI sites is essential. Traditional biological methods for identifying PPIs are costly, labor-intensive, and time-consuming. The development of computational prediction methods for PPI sites offers promising alternatives. Most known deep learning (DL) methods employ layer-wise multi-scale CNNs to extract features from protein sequences. But, these methods usually neglect the spatial positions and hierarchical information embedded within protein sequences, which are actually crucial for PPI site prediction. In this paper, we propose MR2CPPIS, a novel sequence-based DL model that utilizes the multi-scale Res2Net with coordinate attention mechanism to exploit multi-scale features and enhance PPI site prediction capability. We leverage the multi-scale Res2Net to expand the receptive field for each network layer, thus capturing multi-scale information of protein sequences at a granular level. To further explore the local contextual features of each target residue, we employ a coordinate attention block to characterize the precise spatial position information, enabling the network to effectively extract long-range dependencies. We evaluate our MR2CPPIS on three public benchmark datasets (Dset 72, Dset 186, and PDBset 164), achieving state-of-the-art performance. The source codes are available at https://github.com/YyinGong/MR2CPPIS.

Congenital heart disease detection by pediatric electrocardiogram based deep learning integrated with human concepts.

Early detection is critical to achieving improved treatment outcomes for child patients with congenital heart diseases (CHDs). Therefore, developing effective CHD detection techniques using low-cost and non-invasive pediatric electrocardiogram are highly desirable. We propose a deep learning approach for CHD detection, CHDdECG, which automatically extracts features from pediatric electrocardiogram and wavelet transformation characteristics, and integrates them with key human-concept features. Developed on 65,869 cases, CHDdECG achieved ROC-AUC of 0.915 and specificity of 0.881 on a real-world test set covering 12,000 cases. Additionally, on two external test sets with 7137 and 8121 cases, the overall ROC-AUC were 0.917 and 0.907 while specificities were 0.937 and 0.907. Notably, CHDdECG surpassed cardiologists in CHD detection performance comparison, and feature importance scores suggested greater influence of automatically extracted electrocardiogram features on CHD detection compared with human-concept features, implying that CHDdECG may grasp some knowledge beyond human cognition. Our study directly impacts CHD detection with pediatric electrocardiogram and demonstrates the potential of pediatric electrocardiogram for broader benefits.

Mapping Cellular Interactions from Spatially Resolved Transcriptomics Data.

Cell-cell communication (CCC) is essential to how life forms and functions. However, accurate, high-throughput mapping of how expression of all genes in one cell affects expression of all genes in another cell is made possible only recently, through the introduction of spatially resolved transcriptomics technologies (SRTs), especially those that achieve single cell resolution. However, significant challenges remain to analyze such highly complex data properly. Here, we introduce a Bayesian multi-instance learning framework, spacia, to detect CCCs from data generated by SRTs, by uniquely exploiting their spatial modality. We highlight spacia's power to overcome fundamental limitations of popular analytical tools for inference of CCCs, including losing single-cell resolution, limited to ligand-receptor relationships and prior interaction databases, high false positive rates, and most importantly the lack of consideration of the multiple-sender-to-one-receiver paradigm. We evaluated the fitness of spacia for all three commercialized single cell resolution ST technologies: MERSCOPE/Vizgen, CosMx/Nanostring, and Xenium/10X. Spacia unveiled how endothelial cells, fibroblasts and B cells in the tumor microenvironment contribute to Epithelial-Mesenchymal Transition and lineage plasticity in prostate cancer cells. We deployed spacia in a set of pan-cancer datasets and showed that B cells also participate in PDL1/PD1 signaling in tumors. We demonstrated that a CD8+ T cell/PDL1 effectiveness signature derived from spacia analyses is associated with patient survival and response to immune checkpoint inhibitor treatments in 3,354 patients. We revealed differential spatial interaction patterns between γδ T cells and liver hepatocytes in healthy and cancerous contexts. Overall, spacia represents a notable step in advancing quantitative theories of cellular communications.

A Transformer-Based Knowledge Distillation Network for Cortical Cataract Grading.

Cortical cataract, a common type of cataract, is particularly difficult to be diagnosed automatically due to the complex features of the lesions. Recently, many methods based on edge detection or deep learning were proposed for automatic cataract grading. However, these methods suffer a large performance drop in cortical cataract grading due to the more complex cortical opacities and uncertain data. In this paper, we propose a novel Transformer-based Knowledge Distillation Network, called TKD-Net, for cortical cataract grading. To tackle the complex opacity problem, we first devise a zone decomposition strategy to extract more refined features and introduce special sub-scores to consider critical factors of clinical cortical opacity assessment (location, area, density) for comprehensive quantification. Next, we develop a multi-modal mix-attention Transformer to efficiently fuse sub-scores and image modality for complex feature learning. However, obtaining the sub-score modality is a challenge in the clinic, which could cause the modality missing problem instead. To simultaneously alleviate the issues of modality missing and uncertain data, we further design a Transformer-based knowledge distillation method, which uses a teacher model with perfect data to guide a student model with modality-missing and uncertain data. We conduct extensive experiments on a dataset of commonly-used slit-lamp images annotated by the LOCS III grading system to demonstrate that our TKD-Net outperforms state-of-the-art methods, as well as the effectiveness of its key components. Codes are available at https://github.com/wjh892521292/Cataract_TKD-Net.

TestFit: A plug-and-play one-pass test time method for medical image segmentation.

Deep learning (DL) based methods have been extensively studied for medical image segmentation, mostly emphasizing the design and training of DL networks. Only few attempts were made on developing methods for applying DL models in test time. In this paper, we study whether a given off-the-shelf segmentation network can be stably improved on-the-fly during test time in an online processing-and-learning fashion. We propose a new online test-time method, called TestFit, to improve results of a given off-the-shelf DL segmentation model in test time by actively fitting the test data distribution. TestFit first creates a supplementary network (SuppNet) from the given trained off-the-shelf segmentation network (this original network is referred to as OGNet) and applies SuppNet together with OGNet for test time inference. OGNet keeps its hypothesis derived from the original training set to prevent the model from collapsing, while SuppNet seeks to fit the test data distribution. Segmentation results and supervision signals (for updating SuppNet) are generated by combining the outputs of OGNet and SuppNet on the fly. TestFit needs only one pass on each test sample - the same as the traditional test model pipeline - and requires no training time preparation. Since it is challenging to look at only one test sample and no manual annotation for model update each time, we develop a series of technical treatments for improving the stability and effectiveness of our proposed online test-time training method. TestFit works in a plug-and-play fashion, requires minimal hyper-parameter tuning, and is easy to use in practice. Experiments on a large collection of 2D and 3D datasets demonstrate the capability of our TestFit method.

Polygonal Approximation Learning for Convex Object Segmentation in Biomedical Images with Bounding Box Supervision.

As a common and critical medical image analysis task, deep learning based biomedical image segmentation is hindered by the dependence on costly fine-grained annotations. To alleviate this data dependence, in this paper, a novel approach, called Polygonal Approximation Learning (PAL), is proposed for convex object instance segmentation with only bounding-box supervision. The key idea behind PAL is that the detection model for convex objects already contains the necessary information for segmenting them since their convex hulls, which can be generated approximately by the intersection of bounding boxes, are equivalent to the masks representing the objects. To extract the essential information from the detection model, a repeated detection approach is employed on biomedical images where various rotation angles are applied and a dice loss with the projection of the rotated detection results is utilized as a supervised signal in training our segmentation model. In biomedical imaging tasks involving convex objects, such as nuclei instance segmentation, PAL outperforms the known models (e.g., BoxInst) that rely solely on box supervision. Furthermore, PAL achieves comparable performance with mask-supervised models including Mask R-CNN and Cascade Mask R-CNN. Interestingly, PAL also demonstrates remarkable performance on non-convex object instance segmentation tasks, for example, surgical instrument and organ instance segmentation. Our code is available at https://github.com/shenmishajing/PAL.

Embryonic cranial cartilage defects in the Fgfr3Y367C /+ mouse model of achondroplasia.

Achondroplasia, the most common chondrodysplasia in humans, is caused by one of two gain of function mutations localized in the transmembrane domain of fibroblast growth factor receptor 3 (FGFR3) leading to constitutive activation of FGFR3 and subsequent growth plate cartilage and bone defects. Phenotypic features of achondroplasia include macrocephaly with frontal bossing, midface hypoplasia, disproportionate shortening of the extremities, brachydactyly with trident configuration of the hand, and bowed legs. The condition is defined primarily on postnatal effects on bone and cartilage, and embryonic development of tissues in affected individuals is not well studied. Using the Fgfr3Y367C/+ mouse model of achondroplasia, we investigated the developing chondrocranium and Meckel's cartilage (MC) at embryonic days (E)14.5 and E16.5. Sparse hand annotations of chondrocranial and MC cartilages visualized in phosphotungstic acid enhanced three-dimensional (3D) micro-computed tomography (microCT) images were used to train our automatic deep learning-based 3D segmentation model and produce 3D isosurfaces of the chondrocranium and MC. Using 3D coordinates of landmarks measured on the 3D isosurfaces, we quantified differences in the chondrocranium and MC of Fgfr3Y367C/+ mice relative to those of their unaffected littermates. Statistically significant differences in morphology and growth of the chondrocranium and MC were found, indicating direct effects of this Fgfr3 mutation on embryonic cranial and pharyngeal cartilages, which in turn can secondarily affect cranial dermal bone development. Our results support the suggestion that early therapeutic intervention during cartilage formation may lessen the effects of this condition.

A CNN-LSTM Ensemble Model for Predicting Protein-Protein Interaction Binding Sites.

Proteins commonly perform biological functions through protein-protein interactions (PPIs). The knowledge of PPI sites is imperative for the understanding of protein functions, disease mechanisms, and drug design. Traditional biological experimental methods for studying PPI sites still incur considerable drawbacks, including long experimental time and high labor costs. Therefore, many computational methods have been proposed for predicting PPI sites. However, achieving high prediction performance and overcoming severe data imbalance remain challenging issues. In this paper, we propose a new sequence-based deep learning model called CLPPIS (standing for CNN-LSTM ensemble based PPI Sites prediction). CLPPIS consists of CNN and LSTM components, which can capture spatial features and sequential features simultaneously. Further, it utilizes a novel feature group as input, which has 7 physicochemical, biophysical, and statistical properties. Besides, it adopts a batch-weighted loss function to reduce the interference of imbalance data. Our work suggests that the integration of protein spatial features and sequential features provides important information for PPI sites prediction. Evaluation on three public benchmark datasets shows that our CLPPIS model significantly outperforms existing state-of-the-art methods.

Learning from algorithm-generated pseudo-annotations for detecting ants in videos.

Deep learning (DL) based detection models are powerful tools for large-scale analysis of dynamic biological behaviors in video data. Supervised training of a DL detection model often requires a large amount of manually-labeled training data which are time-consuming and labor-intensive to acquire. In this paper, we propose LFAGPA (Learn From Algorithm-Generated Pseudo-Annotations) that utilizes (noisy) annotations which are automatically generated by algorithms to train DL models for ant detection in videos. Our method consists of two main steps: (1) generate foreground objects using a (set of) state-of-the-art foreground extraction algorithm(s); (2) treat the results from step (1) as pseudo-annotations and use them to train deep neural networks for ant detection. We tackle several challenges on how to make use of automatically generated noisy annotations, how to learn from multiple annotation resources, and how to combine algorithm-generated annotations with human-labeled annotations (when available) for this learning framework. In experiments, we evaluate our method using 82 videos (totally 20,348 image frames) captured under natural conditions in a tropical rain-forest for dynamic ant behavior study. Without any manual annotation cost but only algorithm-generated annotations, our method can achieve a decent detection performance (77% in [Formula: see text] score). Moreover, when using only 10% manual annotations, our method can train a DL model to perform as well as using the full human annotations (81% in [Formula: see text] score).

Influence of Data Distribution on Federated Learning Performance in Tumor Segmentation.

Purpose: To investigate the correlation between differences in data distributions and federated deep learning (Fed-DL) algorithm performance in tumor segmentation on CT and MR images. Materials and Methods: Two Fed-DL datasets were retrospectively collected (from November 2020 to December 2021): one dataset of liver tumor CT images (Federated Imaging in Liver Tumor Segmentation [or, FILTS]; three sites, 692 scans) and one publicly available dataset of brain tumor MR images (Federated Tumor Segmentation [or, FeTS]; 23 sites, 1251 scans). Scans from both datasets were grouped according to site, tumor type, tumor size, dataset size, and tumor intensity. To quantify differences in data distributions, the following four distance metrics were calculated: earth mover's distance (EMD), Bhattacharyya distance (BD), χ2 distance (CSD), and Kolmogorov-Smirnov distance (KSD). Both federated and centralized nnU-Net models were trained by using the same grouped datasets. Fed-DL model performance was evaluated by using the ratio of Dice coefficients, θ, between federated and centralized models trained and tested on the same 80:20 split datasets. Results: The Dice coefficient ratio (θ) between federated and centralized models was strongly negatively correlated with the distances between data distributions, with correlation coefficients of -0.920 for EMD, -0.893 for BD, and -0.899 for CSD. However, KSD was weakly correlated with θ, with a correlation coefficient of -0.479. Conclusion: Performance of Fed-DL models in tumor segmentation on CT and MRI datasets was strongly negatively correlated with the distances between data distributions.Keywords: CT, Abdomen/GI, Liver, Comparative Studies, MR Imaging, Brain/Brain Stem, Convolutional Neural Network (CNN), Federated Deep Learning, Tumor Segmentation, Data Distribution Supplemental material is available for this article. © RSNA, 2023See also the commentary by Kwak and Bai in this issue.

CCF-GNN: A Unified Model Aggregating Appearance, Microenvironment, and Topology for Pathology Image Classification.

Pathology images contain rich information of cell appearance, microenvironment, and topology features for cancer analysis and diagnosis. Among such features, topology becomes increasingly important in analysis for cancer immunotherapy. By analyzing geometric and hierarchically structured cell distribution topology, oncologists can identify densely-packed and cancer-relevant cell communities (CCs) for making decisions. Compared to commonly-used pixel-level Convolution Neural Network (CNN) features and cell-instance-level Graph Neural Network (GNN) features, CC topology features are at a higher level of granularity and geometry. However, topological features have not been well exploited by recent deep learning (DL) methods for pathology image classification due to lack of effective topological descriptors for cell distribution and gathering patterns. In this paper, inspired by clinical practice, we analyze and classify pathology images by comprehensively learning cell appearance, microenvironment, and topology in a fine-to-coarse manner. To describe and exploit topology, we design Cell Community Forest (CCF), a novel graph that represents the hierarchical formulation process of big-sparse CCs from small-dense CCs. Using CCF as a new geometric topological descriptor of tumor cells in pathology images, we propose CCF-GNN, a GNN model that successively aggregates heterogeneous features (e.g., appearance, microenvironment) from cell-instance-level, cell-community-level, into image-level for pathology image classification. Extensive cross-validation experiments show that our method significantly outperforms alternative methods on H&E-stained and immunofluorescence images for disease grading tasks with multiple cancer types. Our proposed CCF-GNN establishes a new topological data analysis (TDA) based method, which facilitates integrating multi-level heterogeneous features of point clouds (e.g., for cells) into a unified DL framework.

A new label-free optical imaging method for the lymphatic system enhanced by deep learning.

Our understanding of the lymphatic vascular system lags far behind that of the blood vascular system, limited by available imaging technologies. We present a label-free optical imaging method that visualizes the lymphatic system with high contrast. We developed an orthogonal polarization imaging (OPI) in the shortwave infrared range (SWIR) and imaged both lymph nodes and lymphatic vessels of mice and rats in vivo through intact skin, as well as human mesenteric lymph nodes in colectomy specimens. By integrating SWIR-OPI with U-Net, a deep learning image segmentation algorithm, we automated the lymph node size measurement process. Changes in lymph nodes in response to cancer progression were monitored in two separate mouse cancer models, through which we obtained insights into pre-metastatic niches and correlation between lymph node masses and many important biomarkers. In a human pilot study, we demonstrated the effectiveness of SWIR-OPI to detect human lymph nodes in real time with clinical colectomy specimens. One Sentence Summary: We develop a real-time high contrast optical technique for imaging the lymphatic system, and apply it to anatomical pathology gross examination in a clinical setting, as well as real-time monitoring of tumor microenvironment in animal studies.

TANGO: A GO-Term Embedding Based Method for Protein Semantic Similarity Prediction.

We aim to quantitatively predict protein semantic similarities (PSS), which is vital to making biological discoveries. Previously, researchers commonly exploited Gene Ontology (GO) graphs (containing standardized hierarchically-organized GO terms for annotating distinct protein attributes) to learn GO term embeddings (vector representations) for quantifying protein attribute similarities and aggregate these embeddings to form protein embeddings for similarity measurement. However, two key properties of GO terms and annotated proteins are not yet well-explored by these learning-based methods: (1) taxonomy relations between GO terms; (2) GO terms' different contributions in describing protein semantics. In this paper, we propose TANGO, a new framework composed of a TAxoNomy-aware embedding module and an aggreGatiOn module. Our Embedding Module encodes taxonomic information into GO term embeddings by incorporating GO term topological distances in the GO graph hierarchy. Hence, distances between GO term embeddings can be used to more accurately measure shared meanings between correlated protein attributes. Our Aggregation Module automatically determines the contributions of GO terms when merging into the target protein embeddings, by mining GO term concept dependency relations in the GO graph and correlations in protein annotations. We conduct extensive experiments on several public datasets. On two PSS metrics, our new method significantly outperforms known methods by a large margin.

Identifying Electrocardiogram Abnormalities Using a Handcrafted-Rule-Enhanced Neural Network.

A large number of people suffer from life-threatening cardiac abnormalities, and electrocardiogram (ECG) analysis is beneficial to determining whether an individual is at risk of such abnormalities. Automatic ECG classification methods, especially the deep learning based ones, have been proposed to detect cardiac abnormalities using ECG records, showing good potential to improve clinical diagnosis and help early prevention of cardiovascular diseases. However, the predictions of the known neural networks still do not satisfactorily meet the needs of clinicians, and this phenomenon suggests that some information used in clinical diagnosis may not be well captured and utilized by these methods. In this paper, we introduce some rules into convolutional neural networks, which help present clinical knowledge to deep learning based ECG analysis, in order to improve automated ECG diagnosis performance. Specifically, we propose a Handcrafted-Rule-enhanced Neural Network (called HRNN) for ECG classification with standard 12-lead ECG input, which consists of a rule inference module and a deep learning module. Experiments on two large-scale public ECG datasets show that our new approach considerably outperforms existing state-of-the-art methods. Further, our proposed approach not only can improve the diagnosis performance, but also can assist in detecting mislabelled ECG samples.

KCB-Net: A 3D knee cartilage and bone segmentation network via sparse annotation.

Knee cartilage and bone segmentation is critical for physicians to analyze and diagnose articular damage and knee osteoarthritis (OA). Deep learning (DL) methods for medical image segmentation have largely outperformed traditional methods, but they often need large amounts of annotated data for model training, which is very costly and time-consuming for medical experts, especially on 3D images. In this paper, we report a new knee cartilage and bone segmentation framework, KCB-Net, for 3D MR images based on sparse annotation. KCB-Net selects a small subset of slices from 3D images for annotation, and seeks to bridge the performance gap between sparse annotation and full annotation. Specifically, it first identifies a subset of the most effective and representative slices with an unsupervised scheme; it then trains an ensemble model using the annotated slices; next, it self-trains the model using 3D images containing pseudo-labels generated by the ensemble method and improved by a bi-directional hierarchical earth mover's distance (bi-HEMD) algorithm; finally, it fine-tunes the segmentation results using the primal-dual Internal Point Method (IPM). Experiments on four 3D MR knee joint datasets (the SKI10 dataset, OAI ZIB dataset, Iowa dataset, and iMorphics dataset) show that our new framework outperforms state-of-the-art methods on full annotation, and yields high quality results for small annotation ratios even as low as 10%.

Meckel's Cartilage in Mandibular Development and Dysmorphogenesis.

The Fgfr2c C342Y/+ Crouzon syndrome mouse model carries a cysteine to tyrosine substitution at amino acid position 342 (Cys342Tyr; C342Y) in the fibroblast growth factor receptor 2 (Fgfr2) gene equivalent to a FGFR2 mutation commonly associated with Crouzon and Pfeiffer syndromes in humans. The Fgfr2c C342Y mutation results in constitutive activation of the receptor and is associated with upregulation of osteogenic differentiation. Fgfr2cC342Y/+ Crouzon syndrome mice show premature closure of the coronal suture and other craniofacial anomalies including malocclusion of teeth, most likely due to abnormal craniofacial form. Malformation of the mandible can precipitate a plethora of complications including disrupting development of the upper jaw and palate, impediment of the airway, and alteration of occlusion necessary for proper mastication. The current paradigm of mandibular development assumes that Meckel's cartilage (MC) serves as a support or model for mandibular bone formation and as a template for the later forming mandible. If valid, this implies a functional relationship between MC and the forming mandible, so mandibular dysmorphogenesis might be discerned in MC affecting the relationship between MC and mandibular bone. Here we investigate the relationship of MC to mandible development from the early mineralization of the mandible (E13.5) through the initiation of MC degradation at E17.7 using Fgfr2c C342Y/+ Crouzon syndrome embryos and their unaffected littermates (Fgfr2c +/+ ). Differences between genotypes in both MC and mandibular bone are subtle, however MC of Fgfr2c C342Y/+ embryos is generally longer relative to unaffected littermates at E15.5 with specific aspects remaining relatively large at E17.5. In contrast, mandibular bone is smaller overall in Fgfr2c C342Y/+ embryos relative to their unaffected littermates at E15.5 with the posterior aspect remaining relatively small at E17.5. At a cellular level, differences are identified between genotypes early (E13.5) followed by reduced proliferation in MC (E15.5) and in the forming mandible (E17.5) in Fgfr2c C342Y/+ embryos. Activation of the ERK pathways is reduced in the perichondrium of MC in Fgfr2c C342Y/+ embryos and increased in bone related cells at E15.5. These data reveal that the Fgfr2c C342Y mutation differentially affects cells by type, location, and developmental age indicating a complex set of changes in the cells that make up the lower jaw.

A dysmorphic mouse model reveals developmental interactions of chondrocranium and dermatocranium.

The cranial endo and dermal skeletons, which comprise the vertebrate skull, evolved independently over 470 million years ago and form separately during embryogenesis. In mammals, much of the cartilaginous chondrocranium is transient, undergoing endochondral ossification or disappearing, so its role in skull morphogenesis is not well studied and it remains an enigmatic structure. We provide complete 3D reconstructions of the laboratory mouse chondrocranium from embryonic day (E) 13.5 through E17.5 using a novel methodology of uncertainty-guided segmentation of phosphotungstic enhanced 3D micro-computed tomography images with sparse annotation. We evaluate the embryonic mouse chondrocranium and dermatocranium in 3D, and delineate the effects of a Fgfr2 variant on embryonic chondrocranial cartilages and on their association with forming dermal bones using the Fgfr2cC342Y/+ Crouzon syndrome mouse. We show that the dermatocranium develops outside of and in shapes that conform to the chondrocranium. Results reveal direct effects of the Fgfr2 variant on embryonic cartilage, on chondrocranium morphology, and on the association between chondrocranium and dermatocranium development. Histologically, we observe a trend of relatively more chondrocytes, larger chondrocytes, and/or more matrix in the Fgfr2cC342Y/+ embryos at all timepoints before the chondrocranium begins to disintegrate at E16.5. The chondrocrania and forming dermatocrania of Fgfr2cC342Y/+ embryos are relatively large, but a contrasting trend begins at E16.5 and continues into early postnatal (P0 and P2) timepoints, with the skulls of older Fgfr2cC342Y/+ mice reduced in most dimensions compared to Fgfr2c+/+ littermates. Our findings have implications for the study and treatment of human craniofacial disease, for understanding the impact of chondrocranial morphology on skull growth, and potentially on the evolution of skull morphology.

A Corresponding Region Fusion Framework for Multi-modal Cervical Lesion Detection.

Cervical lesion detection (CLD) using colposcopic images of multi-modality (acetic and iodine) is critical to computer-aided diagnosis (CAD) systems for accurate, objective, and comprehensive cervical cancer screening. To robustly capture lesion features and conform with clinical diagnosis practice, we propose a novel corresponding region fusion network (CRFNet) for multi-modal CLD. CRFNet first extracts feature maps and generates proposals for each modality, then performs proposal shifting to obtain corresponding regions under large position shifts between modalities, and finally fuses those region features with a new corresponding channel attention to detect lesion regions on both modalities. To evaluate CRFNet, we build a large multi-modal colposcopic image dataset collected from our collaborative hospital. We show that our proposed CRFNet surpasses known single-modal and multi-modal CLD methods and achieves state-of-the-art performance, especially in terms of Average Precision.

CMC-Net: 3D calf muscle compartment segmentation with sparse annotation.

Accurate 3D segmentation of calf muscle compartments in volumetric MR images is essential to diagnose as well as assess progression of muscular diseases. Recently, good segmentation performance was achieved using state-of-the-art deep learning approaches, which, however, require large amounts of annotated data for training. Considering that obtaining sufficiently large medical image annotation datasets is often difficult, time-consuming, and requires expert knowledge, minimizing the necessary sizes of expert-annotated training datasets is of great importance. This paper reports CMC-Net, a new deep learning framework for calf muscle compartment segmentation in 3D MR images that selects an effective small subset of 2D slices from the 3D images to be labelled, while also utilizing unannotated slices to facilitate proper generalization of the subsequent training steps. Our model consists of three parts: (1) an unsupervised method to select the most representative 2D slices on which expert annotation is performed; (2) ensemble model training employing these annotated as well as additional unannotated 2D slices; (3) a model-tuning method using pseudo-labels generated by the ensemble model that results in a trained deep network capable of accurate 3D segmentations. Experiments on segmentation of calf muscle compartments in 3D MR images show that our new approach achieves good performance with very small annotation ratios, and when utilizing full annotation, it outperforms state-of-the-art full annotation segmentation methods. Additional experiments on a 3D MR thigh dataset further verify the ability of our method in segmenting leg muscle groups with sparse annotation.